2025-04-02 02:35:07,061 [ 224176 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-04-02 02:35:07,061 [ 224176 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths) 2025-04-02 02:35:07,061 [ 224176 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths) 2025-04-02 02:35:07,061 [ 224176 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:110, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_d43ltr --privileged --dns-search='.' --memory=30709030912 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=8b2301119731 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_keeper_client/test.py::test_four_letter_word_commands test_keeper_client/test.py::test_get_all_children_number test_keeper_client/test.py::test_quoted_argument_parsing test_keeper_client/test.py::test_rm_with_version test_keeper_client/test.py::test_rm_without_version test_keeper_client/test.py::test_set_with_version test_keeper_client/test.py::test_set_without_version test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_three_nodes_two_alive/test.py::test_restart_third_node test_keeper_three_nodes_two_alive/test.py::test_start_non_existing test_keeper_three_nodes_two_alive/test.py::test_start_offline test_ldap_external_user_directory/test.py::test_authentication_fail test_ldap_external_user_directory/test.py::test_authentication_pass test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally test_ldap_external_user_directory/test.py::test_role_mapping test_log_query_probability/test.py::test_log_quries_probability_one test_log_query_probability/test.py::test_log_quries_probability_two test_manipulate_statistics/test.py::test_replicated_db test_manipulate_statistics/test.py::test_replicated_table_ddl test_manipulate_statistics/test.py::test_single_node_normal test_manipulate_statistics/test.py::test_single_node_wide test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation test_named_collections_encrypted/test.py::test_local_storage_encrypted test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster test_non_default_compression/test.py::test_preconfigured_custom_codec test_non_default_compression/test.py::test_preconfigured_default_codec test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec test_non_default_compression/test.py::test_uncompressed_cache_custom_codec test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec test_on_cluster_timeouts/test.py::test_long_query test_optimize_on_insert/test.py::test_empty_parts_optimize 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1]' test_profile_events_s3/test.py::test_profile_events test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_ttl test_quorum_inserts/test.py::test_simple_add_replica test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_time_metric/test.py::test_recovery_time_metric 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False]' 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True]' 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False]' 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True]' -vvv -ss" altinityinfra/integration-tests-runner:2165613c5fcd '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_keeper_client/test.py::test_four_letter_word_commands test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_non_default_compression/test.py::test_preconfigured_custom_codec test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum test_manipulate_statistics/test.py::test_replicated_db test_ldap_external_user_directory/test.py::test_authentication_fail [gw7] PASSED test_manipulate_statistics/test.py::test_replicated_db test_manipulate_statistics/test.py::test_replicated_table_ddl [gw5] PASSED test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part [gw8] PASSED test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart [gw6] PASSED test_non_default_compression/test.py::test_preconfigured_custom_codec test_non_default_compression/test.py::test_preconfigured_default_codec [gw3] PASSED test_keeper_client/test.py::test_four_letter_word_commands test_keeper_client/test.py::test_get_all_children_number [gw3] PASSED test_keeper_client/test.py::test_get_all_children_number test_keeper_client/test.py::test_quoted_argument_parsing [gw3] PASSED test_keeper_client/test.py::test_quoted_argument_parsing test_keeper_client/test.py::test_rm_with_version [gw3] PASSED test_keeper_client/test.py::test_rm_with_version test_keeper_client/test.py::test_rm_without_version [gw3] PASSED test_keeper_client/test.py::test_rm_without_version test_keeper_client/test.py::test_set_with_version [gw3] PASSED test_keeper_client/test.py::test_set_with_version test_keeper_client/test.py::test_set_without_version [gw3] PASSED test_keeper_client/test.py::test_set_without_version [gw5] PASSED test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate test_keeper_session/test.py::test_session_close_shutdown [gw4] PASSED test_ldap_external_user_directory/test.py::test_authentication_fail test_ldap_external_user_directory/test.py::test_authentication_pass [gw4] PASSED test_ldap_external_user_directory/test.py::test_authentication_pass test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw5] PASSED test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw8] PASSED test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart [gw7] PASSED test_manipulate_statistics/test.py::test_replicated_table_ddl test_manipulate_statistics/test.py::test_single_node_normal [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw5] PASSED test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw4] PASSED test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw9] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False] test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw4] PASSED test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally test_ldap_external_user_directory/test.py::test_role_mapping [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] [gw5] PASSED test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw2] PASSED test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] [gw1] SKIPPED test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] [gw4] PASSED test_ldap_external_user_directory/test.py::test_role_mapping [gw2] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] [gw9] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True] test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False] test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster [gw8] PASSED test_keeper_persistent_log/test.py::test_state_duplicate_restart [gw7] PASSED test_manipulate_statistics/test.py::test_single_node_normal test_manipulate_statistics/test.py::test_single_node_wide test_optimize_on_insert/test.py::test_empty_parts_optimize test_keeper_three_nodes_two_alive/test.py::test_restart_third_node [gw2] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] [gw9] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False] test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True] test_named_collections_encrypted/test.py::test_local_storage_encrypted [gw9] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True] test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] [gw7] PASSED test_manipulate_statistics/test.py::test_single_node_wide [gw3] PASSED test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout [gw3] PASSED test_keeper_session/test.py::test_session_timeout test_log_query_probability/test.py::test_log_quries_probability_one test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot [gw6] PASSED test_non_default_compression/test.py::test_preconfigured_default_codec test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] [gw1] PASSED test_optimize_on_insert/test.py::test_empty_parts_optimize [gw8] PASSED test_keeper_three_nodes_two_alive/test.py::test_restart_third_node test_keeper_three_nodes_two_alive/test.py::test_start_non_existing [gw9] PASSED test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_profile_events_s3/test.py::test_profile_events [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] test_recovery_time_metric/test.py::test_recovery_time_metric [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] [gw7] PASSED test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] [gw5] PASSED test_named_collections_encrypted/test.py::test_local_storage_encrypted test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted [gw6] PASSED test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec test_non_default_compression/test.py::test_uncompressed_cache_custom_codec [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] [gw6] PASSED test_non_default_compression/test.py::test_uncompressed_cache_custom_codec test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec [gw6] PASSED test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec [gw1] PASSED test_profile_events_s3/test.py::test_profile_events [gw7] PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw9] PASSED test_recovery_time_metric/test.py::test_recovery_time_metric [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] [gw3] PASSED test_log_query_probability/test.py::test_log_quries_probability_one test_log_query_probability/test.py::test_log_quries_probability_two [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] [gw5] PASSED test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted [gw8] PASSED test_keeper_three_nodes_two_alive/test.py::test_start_non_existing test_keeper_three_nodes_two_alive/test.py::test_start_offline [gw4] PASSED test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] test_on_cluster_timeouts/test.py::test_long_query [gw1] PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions [gw0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] [gw8] PASSED test_keeper_three_nodes_two_alive/test.py::test_start_offline [gw4] PASSED test_on_cluster_timeouts/test.py::test_long_query test_prometheus_endpoint/test.py::test_prometheus_endpoint [gw4] PASSED test_prometheus_endpoint/test.py::test_prometheus_endpoint [gw2] FAILED test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] [gw3] PASSED test_log_query_probability/test.py::test_log_quries_probability_two [gw2] FAILED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] [gw2] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] test_quorum_inserts/test.py::test_insert_quorum_with_ttl [gw2] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_ttl test_quorum_inserts/test.py::test_simple_add_replica [gw2] PASSED test_quorum_inserts/test.py::test_simple_add_replica =================================== FAILURES =================================== ________________ test_insert_quorum_with_keeper_loss_connection ________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_insert_quorum_with_keeper_loss_connection(started_cluster): table_name = "test_insert_quorum_with_keeper_loss_" + uuid.uuid4().hex create_query = ( f"CREATE TABLE {table_name} " "(a Int8, d Date) " "Engine = ReplicatedMergeTree('/clickhouse/tables/{table}', '{replica}') " "ORDER BY a " ) zero.query(create_query) first.query(create_query) first.query(f"SYSTEM STOP FETCHES {table_name}") zero.query("SYSTEM ENABLE FAILPOINT replicated_merge_tree_commit_zk_fail_after_op") zero.query("SYSTEM ENABLE FAILPOINT replicated_merge_tree_insert_retry_pause") with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: insert_future = executor.submit( lambda: zero.query( f"INSERT INTO {table_name}(a,d) VALUES(1, '2011-01-01')", settings={"insert_quorum_timeout": 150000}, ) ) with PartitionManager() as pm: pm.drop_instance_zk_connections(zero) retries = 0 zk = cluster.get_kazoo_client("zoo1") while True: if ( zk.exists( f"/clickhouse/tables/{table_name}/replicas/zero/is_active" ) is None ): break print("replica is still active") time.sleep(1) retries += 1 if retries == 120: raise Exception("Can not wait cluster replica inactive") first.query("SYSTEM ENABLE FAILPOINT finish_set_quorum_failed_parts") quorum_fail_future = executor.submit( lambda: first.query( "SYSTEM WAIT FAILPOINT finish_set_quorum_failed_parts", timeout=300 ) ) first.query(f"SYSTEM START FETCHES {table_name}") concurrent.futures.wait([quorum_fail_future]) > assert quorum_fail_future.exception() is None E AssertionError: assert QueryTimeoutExceedException('Client timed out!') is None E + where QueryTimeoutExceedException('Client timed out!') = >() E + where > = .exception test_quorum_inserts/test.py:411: AssertionError ------------------------------ Captured log call ------------------------------- 2025-04-02 02:35:58 [ 677 ] DEBUG : Executing query CREATE TABLE test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815 (a Int8, d Date) Engine = ReplicatedMergeTree('/clickhouse/tables/{table}', '{replica}') ORDER BY a on zero (cluster.py:3564, query) 2025-04-02 02:35:58 [ 677 ] DEBUG : Executing query CREATE TABLE test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815 (a Int8, d Date) Engine = ReplicatedMergeTree('/clickhouse/tables/{table}', '{replica}') ORDER BY a on first (cluster.py:3564, query) 2025-04-02 02:35:58 [ 677 ] DEBUG : Executing query SYSTEM STOP FETCHES test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815 on first (cluster.py:3564, query) 2025-04-02 02:35:58 [ 677 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT replicated_merge_tree_commit_zk_fail_after_op on zero (cluster.py:3564, query) 2025-04-02 02:35:59 [ 677 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT replicated_merge_tree_insert_retry_pause on zero (cluster.py:3564, query) 2025-04-02 02:35:59 [ 677 ] DEBUG : Executing query INSERT INTO test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815(a,d) VALUES(1, '2011-01-01') on zero (cluster.py:3564, query) 2025-04-02 02:35:59 [ 677 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-02 02:35:59 [ 677 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-02 02:35:59 [ 677 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-02 02:35:59 [ 677 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/containers/create HTTP/1.1" 201 88 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "GET /v1.46/containers/e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/containers/e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a/start HTTP/1.1" 204 0 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : [network] Created new container e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a (network.py:406, _ensure_container) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/containers/e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a/exec HTTP/1.1" 201 74 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/exec/8c63171b6072eb473198fa6274f276a91185fa4621a925b52b9dc07023b384ce/start HTTP/1.1" 101 0 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "GET /v1.46/exec/8c63171b6072eb473198fa6274f276a91185fa4621a925b52b9dc07023b384ce/json HTTP/1.1" 200 460 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : [network] e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a: ['iptables', '--wait', '-I', 'DOCKER-USER', '1', '-p', 'tcp', '-s', '172.16.9.7', '--dport', '2181', '-j', 'DROP'] (0): (network.py:425, _exec_run) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/containers/e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a/exec HTTP/1.1" 201 74 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/exec/b475f2e7958c1da8107bbfbc4a44144ef37302a0368aca0f9e4563971583e640/start HTTP/1.1" 101 0 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "GET /v1.46/exec/b475f2e7958c1da8107bbfbc4a44144ef37302a0368aca0f9e4563971583e640/json HTTP/1.1" 200 460 (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : [network] e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a: ['iptables', '--wait', '-I', 'DOCKER-USER', '1', '-p', 'tcp', '-d', '172.16.9.7', '--sport', '2181', '-j', 'DROP'] (0): (network.py:425, _exec_run) 2025-04-02 02:35:59 [ 677 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2004, get_instance_ip) 2025-04-02 02:35:59 [ 677 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestquoruminserts-gw2-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 02:35:59 [ 677 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.9.2, port:2181, use_ssl:False (cluster.py:3223, get_kazoo_client) 2025-04-02 02:35:59 [ 677 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 02:35:59 [ 677 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-04-02 02:35:59 [ 677 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-04-02 02:35:59 [ 677 ] DEBUG : Sending request(xid=1): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:35:59 [ 677 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:00 [ 677 ] DEBUG : Sending request(xid=2): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:00 [ 677 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:01 [ 677 ] DEBUG : Sending request(xid=3): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:01 [ 677 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:02 [ 677 ] DEBUG : Sending request(xid=4): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:02 [ 677 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:03 [ 677 ] DEBUG : Sending request(xid=5): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:03 [ 677 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:04 [ 677 ] DEBUG : Sending request(xid=6): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:04 [ 677 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:05 [ 677 ] DEBUG : Sending request(xid=7): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:05 [ 677 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:06 [ 677 ] DEBUG : Sending request(xid=8): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:06 [ 677 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:07 [ 677 ] DEBUG : Sending request(xid=9): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:07 [ 677 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:08 [ 677 ] DEBUG : Sending request(xid=10): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:08 [ 677 ] DEBUG : Received response(xid=10): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:09 [ 677 ] DEBUG : Sending request(xid=11): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:09 [ 677 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:10 [ 677 ] DEBUG : Sending request(xid=12): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:10 [ 677 ] DEBUG : Received response(xid=12): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:11 [ 677 ] DEBUG : Sending request(xid=13): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:11 [ 677 ] DEBUG : Received response(xid=13): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:12 [ 677 ] DEBUG : Sending request(xid=14): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:12 [ 677 ] DEBUG : Received response(xid=14): ZnodeStat(czxid=380, mzxid=380, ctime=1743561358272, mtime=1743561358272, version=0, cversion=0, aversion=0, ephemeralOwner=6, dataLength=43, numChildren=0, pzxid=380) (connection.py:410, _read_response) 2025-04-02 02:36:13 [ 677 ] DEBUG : Sending request(xid=15): Exists(path='/clickhouse/tables/test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815/replicas/zero/is_active', watcher=None) (connection.py:312, _submit) 2025-04-02 02:36:13 [ 677 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT finish_set_quorum_failed_parts on first (cluster.py:3564, query) 2025-04-02 02:36:13 [ 677 ] DEBUG : Executing query SYSTEM WAIT FAILPOINT finish_set_quorum_failed_parts on first (cluster.py:3564, query) 2025-04-02 02:36:13 [ 677 ] DEBUG : Executing query SYSTEM START FETCHES test_insert_quorum_with_keeper_loss_d28d6c3f63e44b958f1dce4893c53815 on first (cluster.py:3564, query) 2025-04-02 02:41:14 [ 677 ] DEBUG : Timed out. Last stdout:, stderr: (client.py:242, get_answer) 2025-04-02 02:41:14 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/containers/e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a/exec HTTP/1.1" 201 74 (connectionpool.py:547, _make_request) 2025-04-02 02:41:14 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/exec/851dfdcb16af6921ce6cbcf5608a1d03a47fcfa6a9a9c0aa3a190be3a819b6a9/start HTTP/1.1" 101 0 (connectionpool.py:547, _make_request) 2025-04-02 02:41:14 [ 677 ] DEBUG : http://localhost:None "GET /v1.46/exec/851dfdcb16af6921ce6cbcf5608a1d03a47fcfa6a9a9c0aa3a190be3a819b6a9/json HTTP/1.1" 200 456 (connectionpool.py:547, _make_request) 2025-04-02 02:41:14 [ 677 ] DEBUG : [network] e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a: ['iptables', '--wait', '-D', 'DOCKER-USER', '-p', 'tcp', '-d', '172.16.9.7', '--sport', '2181', '-j', 'DROP'] (0): (network.py:425, _exec_run) 2025-04-02 02:41:14 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/containers/e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a/exec HTTP/1.1" 201 74 (connectionpool.py:547, _make_request) 2025-04-02 02:41:14 [ 677 ] DEBUG : http://localhost:None "POST /v1.46/exec/180adbd4fb76540660dbc798dd5ed84b43c014d6880c77de1e50d37d896c8825/start HTTP/1.1" 101 0 (connectionpool.py:547, _make_request) 2025-04-02 02:41:14 [ 677 ] DEBUG : http://localhost:None "GET /v1.46/exec/180adbd4fb76540660dbc798dd5ed84b43c014d6880c77de1e50d37d896c8825/json HTTP/1.1" 200 456 (connectionpool.py:547, _make_request) 2025-04-02 02:41:14 [ 677 ] DEBUG : [network] e854a350157aee7f9799099c6c7537db2497005620942dbf510be4a3631c564a: ['iptables', '--wait', '-D', 'DOCKER-USER', '-p', 'tcp', '-s', '172.16.9.7', '--dport', '2181', '-j', 'DROP'] (0): (network.py:425, _exec_run) ________________ test_insert_quorum_with_move_partition[False] _________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = add_new_data = False @pytest.mark.parametrize(("add_new_data"), [False, True]) def test_insert_quorum_with_move_partition(started_cluster, add_new_data): # use different table names for easier disambiguation in logs between runs (you may also check uuid though, but not always convenient) source_table_name = ( "test_insert_quorum_with_move_partition_source_new_data" if add_new_data else "test_insert_quorum_with_move_partition_source" ) + uuid.uuid4().hex destination_table_name = ( "test_insert_quorum_with_move_partition_destination_new_data" if add_new_data else "test_insert_quorum_with_move_partition_destination" ) + uuid.uuid4().hex zero.query(f"DROP TABLE IF EXISTS {source_table_name} ON CLUSTER cluster") zero.query(f"DROP TABLE IF EXISTS {destination_table_name} ON CLUSTER cluster") create_source = ( f"CREATE TABLE {source_table_name} ON CLUSTER cluster " "(a Int8, d Date) " "Engine = ReplicatedMergeTree " "PARTITION BY d ORDER BY a " ) create_destination = ( f"CREATE TABLE {destination_table_name} ON CLUSTER cluster " "(a Int8, d Date) " "Engine = ReplicatedMergeTree " "PARTITION BY d ORDER BY a " ) print("Create source Replicated table with three replicas") zero.query(create_source) print("Create destination Replicated table with three replicas") zero.query(create_destination) print(f"Stop fetches for {source_table_name} at first replica.") first.query(f"SYSTEM STOP FETCHES {source_table_name}") print("Insert with quorum. (zero and second)") > zero.query(f"INSERT INTO {source_table_name}(a,d) VALUES(1, '2011-01-01')") test_quorum_inserts/test.py:242: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3565: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() helpers/client.py:230: in get_answer self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) /usr/lib/python3.10/subprocess.py:1209: in wait return self._wait(timeout=timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = timeout = 600 def _wait(self, timeout): """Internal implementation of wait() on POSIX.""" if self.returncode is not None: return self.returncode if timeout is not None: endtime = _time() + timeout # Enter a busy loop if we have a timeout. This busy loop was # cribbed from Lib/threading.py in Thread.wait() at r71065. delay = 0.0005 # 500 us -> initial delay of 1 ms while True: if self._waitpid_lock.acquire(False): try: if self.returncode is not None: break # Another thread waited. (pid, sts) = self._try_wait(os.WNOHANG) assert pid == self.pid or pid == 0 if pid == self.pid: self._handle_exitstatus(sts) break finally: self._waitpid_lock.release() remaining = self._remaining_time(endtime) if remaining <= 0: > raise TimeoutExpired(self.args, timeout) E subprocess.TimeoutExpired: Command '['/clickhouse', 'client', '--host', '172.16.9.7', '--port', '9000', '--stacktrace']' timed out after 600 seconds /usr/lib/python3.10/subprocess.py:1951: TimeoutExpired ------------------------------ Captured log call ------------------------------- 2025-04-02 02:41:14 [ 677 ] DEBUG : Executing query DROP TABLE IF EXISTS test_insert_quorum_with_move_partition_source8be3d03babcc4e7a98c0df2e07a824a3 ON CLUSTER cluster on zero (cluster.py:3564, query) 2025-04-02 02:41:14 [ 677 ] DEBUG : Executing query DROP TABLE IF EXISTS test_insert_quorum_with_move_partition_destination2b50b79344b545b6bef63c0f1508180a ON CLUSTER cluster on zero (cluster.py:3564, query) 2025-04-02 02:41:15 [ 677 ] DEBUG : Executing query CREATE TABLE test_insert_quorum_with_move_partition_source8be3d03babcc4e7a98c0df2e07a824a3 ON CLUSTER cluster (a Int8, d Date) Engine = ReplicatedMergeTree PARTITION BY d ORDER BY a on zero (cluster.py:3564, query) 2025-04-02 02:41:15 [ 677 ] DEBUG : Executing query CREATE TABLE test_insert_quorum_with_move_partition_destination2b50b79344b545b6bef63c0f1508180a ON CLUSTER cluster (a Int8, d Date) Engine = ReplicatedMergeTree PARTITION BY d ORDER BY a on zero (cluster.py:3564, query) 2025-04-02 02:41:15 [ 677 ] DEBUG : Executing query SYSTEM STOP FETCHES test_insert_quorum_with_move_partition_source8be3d03babcc4e7a98c0df2e07a824a3 on first (cluster.py:3564, query) 2025-04-02 02:41:16 [ 677 ] DEBUG : Executing query INSERT INTO test_insert_quorum_with_move_partition_source8be3d03babcc4e7a98c0df2e07a824a3(a,d) VALUES(1, '2011-01-01') on zero (cluster.py:3564, query) ============================== slowest durations =============================== 602.45s call test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] 345.50s call test_log_query_probability/test.py::test_log_quries_probability_two 316.16s call test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 53.67s call test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster 47.70s call test_keeper_three_nodes_two_alive/test.py::test_start_non_existing 42.88s call test_keeper_three_nodes_two_alive/test.py::test_start_offline 38.20s call test_non_default_compression/test.py::test_preconfigured_default_codec 34.31s call test_log_query_probability/test.py::test_log_quries_probability_one 30.68s setup test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False] 29.72s call test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec 27.97s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 27.22s call test_on_cluster_timeouts/test.py::test_long_query 27.15s setup test_ldap_external_user_directory/test.py::test_authentication_fail 26.95s setup test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 25.65s call test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted 23.79s setup test_named_collections_encrypted/test.py::test_local_storage_encrypted 22.84s setup test_keeper_client/test.py::test_four_letter_word_commands 22.57s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] 21.81s teardown test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted 21.62s teardown test_profile_events_s3/test.py::test_profile_events 20.41s teardown test_quorum_inserts/test.py::test_simple_add_replica 20.37s setup test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster 20.26s setup test_on_cluster_timeouts/test.py::test_long_query 20.00s teardown test_on_cluster_timeouts/test.py::test_long_query 19.06s call test_quorum_inserts/test.py::test_insert_quorum_with_ttl 18.65s setup test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part 18.22s setup test_manipulate_statistics/test.py::test_replicated_db 17.36s setup test_profile_events_s3/test.py::test_profile_events 17.19s teardown test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation 16.96s teardown test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster 16.91s setup test_optimize_on_insert/test.py::test_empty_parts_optimize 16.06s setup test_keeper_persistent_log/test.py::test_ephemeral_after_restart 15.94s setup test_non_default_compression/test.py::test_preconfigured_custom_codec 15.92s setup test_keeper_session/test.py::test_session_close_shutdown 14.66s setup test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order 13.93s setup test_read_temporary_tables_on_failure/test.py::test_different_versions 13.86s setup test_prometheus_endpoint/test.py::test_prometheus_endpoint 13.59s setup test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 13.52s call test_keeper_session/test.py::test_session_close_shutdown 13.43s call test_recovery_time_metric/test.py::test_recovery_time_metric 13.20s call test_manipulate_statistics/test.py::test_single_node_normal 13.04s setup test_keeper_three_nodes_two_alive/test.py::test_restart_third_node 12.56s call test_keeper_persistent_log/test.py::test_state_duplicate_restart 11.62s call test_manipulate_statistics/test.py::test_single_node_wide 10.94s setup test_log_query_probability/test.py::test_log_quries_probability_one 10.60s setup test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot 10.23s call test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot 9.86s setup test_recovery_time_metric/test.py::test_recovery_time_metric 9.80s call test_named_collections_encrypted/test.py::test_local_storage_encrypted 8.91s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] 8.73s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] 8.47s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] 8.29s teardown test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec 8.28s call test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 8.20s call test_manipulate_statistics/test.py::test_replicated_table_ddl 7.85s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] 7.78s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] 7.72s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 7.59s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] 7.56s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] 7.54s call test_profile_events_s3/test.py::test_profile_events 7.38s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] 7.26s call test_keeper_three_nodes_two_alive/test.py::test_restart_third_node 7.25s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] 6.99s call test_keeper_persistent_log/test.py::test_state_after_restart 6.88s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] 6.72s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] 6.62s call test_non_default_compression/test.py::test_preconfigured_custom_codec 6.48s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] 6.21s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] 6.21s call test_keeper_persistent_log/test.py::test_ephemeral_after_restart 6.01s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] 5.92s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] 5.89s teardown test_optimize_on_insert/test.py::test_empty_parts_optimize 5.86s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] 5.79s teardown test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order 5.77s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] 5.63s teardown test_log_query_probability/test.py::test_log_quries_probability_two 5.50s call test_read_temporary_tables_on_failure/test.py::test_different_versions 5.26s teardown test_manipulate_statistics/test.py::test_single_node_wide 5.05s call test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] 4.85s call test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] 4.67s teardown test_recovery_time_metric/test.py::test_recovery_time_metric 4.52s call test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True] 4.35s call test_quorum_inserts/test.py::test_simple_add_replica 4.30s call test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes 4.22s call test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] 4.15s teardown test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 3.64s call test_ldap_external_user_directory/test.py::test_role_mapping 3.64s teardown test_read_temporary_tables_on_failure/test.py::test_different_versions 3.52s call test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate 3.50s call test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part 3.39s call test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True] 3.34s call test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge 3.19s call test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False] 3.14s call test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False] 3.12s call test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation 2.98s teardown test_ldap_external_user_directory/test.py::test_role_mapping 2.94s call test_optimize_on_insert/test.py::test_empty_parts_optimize 2.92s call test_manipulate_statistics/test.py::test_replicated_db 2.88s setup test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False] 2.79s teardown test_prometheus_endpoint/test.py::test_prometheus_endpoint 2.70s setup test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True] 2.68s call test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part 2.41s setup test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True] 2.40s teardown test_keeper_client/test.py::test_set_without_version 2.26s call test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally 1.75s teardown test_keeper_persistent_log/test.py::test_state_duplicate_restart 1.69s teardown test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot 1.55s teardown test_keeper_session/test.py::test_session_timeout 1.45s teardown test_keeper_three_nodes_two_alive/test.py::test_start_offline 1.35s call test_non_default_compression/test.py::test_uncompressed_cache_custom_codec 1.25s call test_prometheus_endpoint/test.py::test_prometheus_endpoint 1.17s call test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec 0.80s call test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 0.64s call test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order 0.53s call test_ldap_external_user_directory/test.py::test_authentication_fail 0.37s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.37s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.35s call test_keeper_session/test.py::test_session_timeout 0.34s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.33s call test_ldap_external_user_directory/test.py::test_authentication_pass 0.33s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.33s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.32s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.28s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.27s call test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.15s setup test_keeper_client/test.py::test_rm_with_version 0.14s setup test_keeper_client/test.py::test_set_with_version 0.13s setup test_keeper_client/test.py::test_set_without_version 0.12s setup test_keeper_client/test.py::test_rm_without_version 0.11s setup test_keeper_client/test.py::test_quoted_argument_parsing 0.10s setup test_keeper_client/test.py::test_get_all_children_number 0.05s call test_keeper_client/test.py::test_get_all_children_number 0.04s teardown test_keeper_client/test.py::test_rm_with_version 0.03s teardown test_keeper_client/test.py::test_set_with_version 0.03s call test_keeper_client/test.py::test_rm_with_version 0.03s call test_keeper_client/test.py::test_quoted_argument_parsing 0.03s teardown test_keeper_client/test.py::test_get_all_children_number 0.03s teardown test_keeper_client/test.py::test_quoted_argument_parsing 0.02s teardown test_keeper_client/test.py::test_rm_without_version 0.02s teardown test_keeper_client/test.py::test_four_letter_word_commands 0.02s call test_keeper_client/test.py::test_set_with_version 0.02s call test_keeper_client/test.py::test_set_without_version 0.01s call test_keeper_client/test.py::test_rm_without_version 0.00s call test_keeper_client/test.py::test_four_letter_word_commands 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_keeper_three_nodes_two_alive/test.py::test_restart_third_node 0.00s teardown test_keeper_persistent_log/test.py::test_ephemeral_after_restart 0.00s teardown test_ldap_external_user_directory/test.py::test_authentication_fail 0.00s teardown test_keeper_persistent_log/test.py::test_state_after_restart 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_keeper_session/test.py::test_session_close_shutdown 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] 0.00s teardown test_log_query_probability/test.py::test_log_quries_probability_one 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] 0.00s teardown test_named_collections_encrypted/test.py::test_local_storage_encrypted 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] 0.00s teardown test_keeper_three_nodes_two_alive/test.py::test_start_non_existing 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] 0.00s setup test_keeper_three_nodes_two_alive/test.py::test_start_non_existing 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_manipulate_statistics/test.py::test_replicated_db 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] 0.00s teardown test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True] 0.00s setup test_quorum_inserts/test.py::test_simple_add_replica 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] 0.00s setup test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_manipulate_statistics/test.py::test_single_node_normal 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] 0.00s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] 0.00s setup test_manipulate_statistics/test.py::test_single_node_wide 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] 0.00s teardown test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part 0.00s teardown test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_keeper_session/test.py::test_session_timeout 0.00s setup test_non_default_compression/test.py::test_uncompressed_cache_custom_codec 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge 0.00s setup test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate 0.00s teardown test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] 0.00s teardown test_non_default_compression/test.py::test_preconfigured_custom_codec 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 0.00s setup test_log_query_probability/test.py::test_log_quries_probability_two 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_ttl 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] 0.00s setup test_keeper_three_nodes_two_alive/test.py::test_start_offline 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] 0.00s teardown test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part 0.00s setup test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec 0.00s setup test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_keeper_persistent_log/test.py::test_state_after_restart 0.00s setup test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_ttl 0.00s setup test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge 0.00s teardown test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] 0.00s setup test_keeper_persistent_log/test.py::test_state_duplicate_restart 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] 0.00s setup test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation 0.00s setup test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] 0.00s setup test_manipulate_statistics/test.py::test_single_node_normal 0.00s setup test_ldap_external_user_directory/test.py::test_authentication_pass 0.00s setup test_non_default_compression/test.py::test_preconfigured_default_codec 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s setup test_manipulate_statistics/test.py::test_replicated_table_ddl 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] 0.00s teardown test_non_default_compression/test.py::test_preconfigured_default_codec 0.00s setup test_ldap_external_user_directory/test.py::test_role_mapping 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_manipulate_statistics/test.py::test_replicated_table_ddl 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}] 0.00s teardown test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}] 0.00s teardown test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] 0.00s teardown test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes 0.00s teardown test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally 0.00s teardown test_non_default_compression/test.py::test_uncompressed_cache_custom_codec 0.00s teardown test_ldap_external_user_directory/test.py::test_authentication_pass =========================== short test summary info ============================ FAILED test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection FAILED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] PASSED test_manipulate_statistics/test.py::test_replicated_db PASSED test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part PASSED test_keeper_persistent_log/test.py::test_ephemeral_after_restart PASSED test_non_default_compression/test.py::test_preconfigured_custom_codec PASSED test_keeper_client/test.py::test_four_letter_word_commands PASSED test_keeper_client/test.py::test_get_all_children_number PASSED test_keeper_client/test.py::test_quoted_argument_parsing PASSED test_keeper_client/test.py::test_rm_with_version PASSED test_keeper_client/test.py::test_rm_without_version PASSED test_keeper_client/test.py::test_set_with_version PASSED test_keeper_client/test.py::test_set_without_version PASSED test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part PASSED test_ldap_external_user_directory/test.py::test_authentication_fail PASSED test_ldap_external_user_directory/test.py::test_authentication_pass PASSED test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate PASSED test_keeper_persistent_log/test.py::test_state_after_restart PASSED test_manipulate_statistics/test.py::test_replicated_table_ddl PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0] PASSED test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge PASSED test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False] PASSED test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally PASSED test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation PASSED test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1] PASSED test_ldap_external_user_directory/test.py::test_role_mapping PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True] PASSED test_keeper_persistent_log/test.py::test_state_duplicate_restart PASSED test_manipulate_statistics/test.py::test_single_node_normal PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False] PASSED test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1] PASSED test_manipulate_statistics/test.py::test_single_node_wide PASSED test_keeper_session/test.py::test_session_close_shutdown PASSED test_keeper_session/test.py::test_session_timeout PASSED test_non_default_compression/test.py::test_preconfigured_default_codec PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0] PASSED test_optimize_on_insert/test.py::test_empty_parts_optimize PASSED test_keeper_three_nodes_two_alive/test.py::test_restart_third_node PASSED test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0] PASSED test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1] PASSED test_named_collections_encrypted/test.py::test_local_storage_encrypted PASSED test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0] PASSED test_non_default_compression/test.py::test_uncompressed_cache_custom_codec PASSED test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec PASSED test_profile_events_s3/test.py::test_profile_events PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict PASSED test_recovery_time_metric/test.py::test_recovery_time_metric PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1] PASSED test_log_query_probability/test.py::test_log_quries_probability_one PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1] PASSED test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted PASSED test_keeper_three_nodes_two_alive/test.py::test_start_non_existing PASSED test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1] PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0] PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions PASSED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] PASSED test_keeper_three_nodes_two_alive/test.py::test_start_offline PASSED test_on_cluster_timeouts/test.py::test_long_query PASSED test_prometheus_endpoint/test.py::test_prometheus_endpoint PASSED test_log_query_probability/test.py::test_log_quries_probability_two PASSED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_ttl PASSED test_quorum_inserts/test.py::test_simple_add_replica SKIPPED [24] test_parallel_replicas_all_marks_read/test.py:137: Disabled for sanitizers (too slow) ============ 2 failed, 74 passed, 24 skipped in 1023.92s (0:17:03) ============= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 528, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_d43ltr --privileged --dns-search='.' --memory=30709030912 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=8b2301119731 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_keeper_client/test.py::test_four_letter_word_commands test_keeper_client/test.py::test_get_all_children_number test_keeper_client/test.py::test_quoted_argument_parsing test_keeper_client/test.py::test_rm_with_version test_keeper_client/test.py::test_rm_without_version test_keeper_client/test.py::test_set_with_version test_keeper_client/test.py::test_set_without_version test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart test_keeper_restore_from_snapshot/test.py::test_recover_from_snapshot test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_three_nodes_two_alive/test.py::test_restart_third_node test_keeper_three_nodes_two_alive/test.py::test_start_non_existing test_keeper_three_nodes_two_alive/test.py::test_start_offline test_ldap_external_user_directory/test.py::test_authentication_fail test_ldap_external_user_directory/test.py::test_authentication_pass test_ldap_external_user_directory/test.py::test_push_role_to_other_nodes test_ldap_external_user_directory/test.py::test_remote_query_user_does_not_exist_locally test_ldap_external_user_directory/test.py::test_role_mapping test_log_query_probability/test.py::test_log_quries_probability_one test_log_query_probability/test.py::test_log_quries_probability_two test_manipulate_statistics/test.py::test_replicated_db test_manipulate_statistics/test.py::test_replicated_table_ddl test_manipulate_statistics/test.py::test_single_node_normal test_manipulate_statistics/test.py::test_single_node_wide test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_random_part test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part test_max_suspicious_broken_parts_replicated/test.py::test_corrupted_unexpected_part_ultimate test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_merge test_max_suspicious_broken_parts_replicated/test.py::test_unexpected_uncommitted_mutation test_named_collections_encrypted/test.py::test_local_storage_encrypted test_named_collections_encrypted/test.py::test_zookeper_storage_encrypted test_named_collections_if_exists_on_cluster/test.py::test_create_alter_drop_on_cluster test_non_default_compression/test.py::test_preconfigured_custom_codec test_non_default_compression/test.py::test_preconfigured_default_codec test_non_default_compression/test.py::test_preconfigured_deflateqpl_codec test_non_default_compression/test.py::test_uncompressed_cache_custom_codec test_non_default_compression/test.py::test_uncompressed_cache_plus_zstd_codec test_on_cluster_timeouts/test.py::test_long_query test_optimize_on_insert/test.py::test_empty_parts_optimize 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[1-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-10-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-1000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-10000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name} WHERE intDiv(key, 100) IN {rand_set}]' 'test_parallel_replicas_all_marks_read/test.py::test_number_of_marks_read[10-100-100000-SELECT sum(cityHash64(*)) FROM {table_name}]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_multiple_shards_multiple_replicas-4-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-10-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-2-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-3-1]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-0]' 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1]' test_profile_events_s3/test.py::test_profile_events test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_ttl test_quorum_inserts/test.py::test_simple_add_replica test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_time_metric/test.py::test_recovery_time_metric 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-False]' 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-False-True]' 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-False]' 'test_refreshable_mat_view/test.py::test_alters[settings0-None-False-None-False-True-True]' -vvv -ss" altinityinfra/integration-tests-runner:2165613c5fcd ' returned non-zero exit status 1.